Distributed Value Function Approximation for Collaborative Multiagent Reinforcement Learning

In this article, we propose several novel distributed gradient-based temporal-difference algorithms for multiagent off-policy learning of linear approximation the value function in Markov decision processes with strict information structure constraints, limiting interagent communications to small neighborhoods. The are composed following: first, local parameter updates based on single-agent gradient algorithms, including eligibility traces state-dependent parameters and, second, stochastic time-varying consensus schemes, represented by directed graphs. proposed differ their form, definition traces, selection time scales, and way incorporating iterations. main contribution article is a convergence analysis general properties underlying Feller-Markov model. We prove under assumptions that estimates generated all weakly converge corresponding ordinary differential equations precisely defined invariant sets. It demonstrated how adopted methodology can be applied weaker constraints. variance reduction effect formulating analyzing an asymptotic equation. Specific guidelines communication network design provided. algorithms' superior illustrated characteristic simulation results.